MHD-Net: Memory-Aware Hetero-Modal Distillation Network for Thymic Epithelial Tumor Typing With Missing Pathology Modality.

Fusing multi-modal radiology and pathology data with complementary information can improve the accuracy of tumor typing. However, collecting pathology data is difficult since it is high-cost and sometimes only obtainable after the surgery, which limits the application of multi-modal methods in diagnosis. To address this problem, we propose comprehensively learning multi-modal radiology-pathology data in training, and only using uni-modal radiology data in testing. Concretely, a Memory-aware Hetero-modal Distillation Network (MHD-Net) is proposed, which can distill well-learned multi-modal knowledge with the assistance of memory from the teacher to the student. In the teacher, to tackle the challenge in hetero-modal feature fusion, we propose a novel spatial-differentiated hetero-modal fusion module (SHFM) that models spatial-specific tumor information correlations across modalities. As only radiology data is accessible to the student, we store pathology features in the proposed contrast-boosted typing memory module (CTMM) that achieves type-wise memory updating and stage-wise contrastive memory boosting to ensure the effectiveness and generalization of memory items. In the student, to improve the cross-modal distillation, we propose a multi-stage memory-aware distillation (MMD) scheme that reads memory-aware pathology features from CTMM to remedy missing modal-specific information. Furthermore, we construct a Radiology-Pathology Thymic Epithelial Tumor (RPTET) dataset containing paired CT and WSI images with annotations. Experiments on the RPTET and CPTAC-LUAD datasets demonstrate that MHD-Net significantly improves tumor typing and outperforms existing multi-modal methods on missing modality situations.

Integrated lithium niobate microwave photonic processing engine.

Integrated microwave photonics (MWP) is an intriguing technology for the generation, transmission and manipulation of microwave signals in chip-scale optical systems1,2. In particular, ultrafast processing of analogue signals in the optical domain with high fidelity and low latency could enable a variety of applications such as MWP filters3-5, microwave signal processing6-9 and image recognition10,11. An ideal integrated MWP processing platform should have both an efficient and high-speed electro-optic modulation block to faithfully perform microwave-optic conversion at low power and also a low-loss functional photonic network to implement various signal-processing tasks. Moreover, large-scale, low-cost manufacturability is required to monolithically integrate the two building blocks on the same chip. Here we demonstrate such an integrated MWP processing engine based on a 4 inch wafer-scale thin-film lithium niobate platform. It can perform multipurpose tasks with processing bandwidths of up to 67 GHz at complementary metal-oxide-semiconductor (CMOS)-compatible voltages. We achieve ultrafast analogue computation, namely temporal integration and differentiation, at sampling rates of up to 256 giga samples per second, and deploy these functions to showcase three proof-of-concept applications: solving ordinary differential equations, generating ultra-wideband signals and detecting edges in images. We further leverage the image edge detector to realize a photonic-assisted image segmentation model that can effectively outline the boundaries of melanoma lesion in medical diagnostic images. Our ultrafast lithium niobate MWP engine could provide compact, low-latency and cost-effective solutions for future wireless communications, high-resolution radar and photonic artificial intelligence.

Bioactive Polyurethane-Poly(ethylene Glycol) Diacrylate Hydrogels for Applications in Tissue Engineering.

Polyurethanes (PUs) are a highly adaptable class of biomaterials that are among some of the most researched materials for various biomedical applications. However, engineered tissue scaffolds composed of PU have not found their way into clinical application, mainly due to the difficulty of balancing the control of material properties with the desired cellular response. A simple method for the synthesis of tunable bioactive poly(ethylene glycol) diacrylate (PEGDA) hydrogels containing photocurable PU is described. These hydrogels may be modified with PEGylated peptides or proteins to impart variable biological functions, and the mechanical properties of the hydrogels can be tuned based on the ratios of PU and PEGDA. Studies with human cells revealed that PU-PEG blended hydrogels support cell adhesion and viability when cell adhesion peptides are crosslinked within the hydrogel matrix. These hydrogels represent a unique and highly tailorable system for synthesizing PU-based synthetic extracellular matrices for tissue engineering applications.

MGIML: Cancer Grading with Incomplete Radiology-Pathology Data via Memory Learning and Gradient Homogenization.

Taking advantage of multi-modal radiology-pathology data with complementary clinical information for cancer grading is helpful for doctors to improve diagnosis efficiency and accuracy. However, radiology and pathology data have distinct acquisition difficulties and costs, which leads to incomplete-modality data being common in applications. In this work, we propose a Memory-and Gradient-guided Incomplete Modal-modal Learning (MGIML) framework for cancer grading with incomplete radiology-pathology data. Firstly, to remedy missing-modality information, we propose a Memory-driven Hetero-modality Complement (MH-Complete) scheme, which constructs modal-specific memory banks constrained by a coarse-grained memory boosting (CMB) loss to record generic radiology and pathology feature patterns, and develops a cross-modal memory reading strategy enhanced by a fine-grained memory consistency (FMC) loss to take missing-modality information from well-stored memories. Secondly, as gradient conflicts exist between missing-modality situations, we propose a Rotation-driven Gradient Homogenization (RG-Homogenize) scheme, which estimates instance-specific rotation matrices to smoothly change the feature-level gradient directions, and computes confidence-guided homogenization weights to dynamically balance gradient magnitudes. By simultaneously mitigating gradient direction and magnitude conflicts, this scheme well avoids the negative transfer and optimization imbalance problems. Extensive experiments on CPTAC-UCEC and CPTAC-PDA datasets show that the proposed MGIML framework performs favorably against state-of-the-art multi-modal methods on missing-modality situations.

Comprehensive learning and adaptive teaching: Distilling multi-modal knowledge for pathological glioma grading.

The fusion of multi-modal data, e.g., pathology slides and genomic profiles, can provide complementary information and benefit glioma grading. However, genomic profiles are difficult to obtain due to the high costs and technical challenges, thus limiting the clinical applications of multi-modal diagnosis. In this work, we investigate the realistic problem where paired pathology-genomic data are available during training, while only pathology slides are accessible for inference. To solve this problem, a comprehensive learning and adaptive teaching framework is proposed to improve the performance of pathological grading models by transferring the privileged knowledge from the multi-modal teacher to the pathology student. For comprehensive learning of the multi-modal teacher, we propose a novel Saliency-Aware Masking (SA-Mask) strategy to explore richer disease-related features from both modalities by masking the most salient features. For adaptive teaching of the pathology student, we first devise a Local Topology Preserving and Discrepancy Eliminating Contrastive Distillation (TDC-Distill) module to align the feature distributions of the teacher and student models. Furthermore, considering the multi-modal teacher may include incorrect information, we propose a Gradient-guided Knowledge Refinement (GK-Refine) module that builds a knowledge bank and adaptively absorbs the reliable knowledge according to their agreement in the gradient space. Experiments on the TCGA GBM-LGG dataset show that our proposed distillation framework improves the pathological glioma grading and outperforms other KD methods. Notably, with the sole pathology slides, our method achieves comparable performance with existing multi-modal methods. The code is available at https://github.com/CUHK-AIM-Group/MultiModal-learning.

Gradient modulated contrastive distillation of low-rank multi-modal knowledge for disease diagnosis.

The fusion of multi-modal data, e.g., medical images and genomic profiles, can provide complementary information and further benefit disease diagnosis. However, multi-modal disease diagnosis confronts two challenges: (1) how to produce discriminative multi-modal representations by exploiting complementary information while avoiding noisy features from different modalities. (2) how to obtain an accurate diagnosis when only a single modality is available in real clinical scenarios. To tackle these two issues, we present a two-stage disease diagnostic framework. In the first multi-modal learning stage, we propose a novel Momentum-enriched Multi-Modal Low-Rank (M3LR) constraint to explore the high-order correlations and complementary information among different modalities, thus yielding more accurate multi-modal diagnosis. In the second stage, the privileged knowledge of the multi-modal teacher is transferred to the unimodal student via our proposed Discrepancy Supervised Contrastive Distillation (DSCD) and Gradient-guided Knowledge Modulation (GKM) modules, which benefit the unimodal-based diagnosis. We have validated our approach on two tasks: (i) glioma grading based on pathology slides and genomic data, and (ii) skin lesion classification based on dermoscopy and clinical images. Experimental results on both tasks demonstrate that our proposed method consistently outperforms existing approaches in both multi-modal and unimodal diagnoses.

Dynamic Depth-Aware Network for Endoscopy Super-Resolution.

Endoscopy super-resolution (SR) plays an important role in improving diagnostic results and reducing the misdiagnosis rate. Even though recent studies have investigated the SR for endoscopy, these methods apply equal importance to the whole image and do not consider the relationship among pixels, especially the depth information, which can provide diagnosis-related information for clinicians. To address this problem, we propose a dynamic depth-aware network for endoscopy super-resolution, which represents the first effort to comprehensively integrate the depth information to the SR task for endoscopic images. It includes a depth-wise feature extracting branch (DW-B) and a depth-guided SR branch (DGSR-B). The DW-B aims to extract the representative feature for each depth level (i.e. depth matrix) further to provide auxiliary information and guide the super-resolution of texture under different depth levels. In DGSR-B, a depth-guided block (DGB) consisting of depth-focus normalization (DFN) is introduced to inject both the depth matrix and depth map into the LR image feature, so as to guide the image generation for each depth region. To adaptively super-resolve the regions under different depth levels, we devise a dynamic depth-aware loss to assign different trainable weights to each region for SR optimization. Extensive experiments have been conducted on two main publicly available datasets, i.e., the Kvasir dataset and the EndoScene dataset, and the superior performance verifies the effectiveness of our method for SR task and polyp segmentation. Source code is to be released.

D2-Net: Dual Disentanglement Network for Brain Tumor Segmentation With Missing Modalities.

Multi-modal Magnetic Resonance Imaging (MRI) can provide complementary information for automatic brain tumor segmentation, which is crucial for diagnosis and prognosis. While missing modality data is common in clinical practice and it can result in the collapse of most previous methods relying on complete modality data. Current state-of-the-art approaches cope with the situations of missing modalities by fusing multi-modal images and features to learn shared representations of tumor regions, which often ignore explicitly capturing the correlations among modalities and tumor regions. Inspired by the fact that modality information plays distinct roles to segment different tumor regions, we aim to explicitly exploit the correlations among various modality-specific information and tumor-specific knowledge for segmentation. To this end, we propose a Dual Disentanglement Network (D2-Net) for brain tumor segmentation with missing modalities, which consists of a modality disentanglement stage (MD-Stage) and a tumor-region disentanglement stage (TD-Stage). In the MD-Stage, a spatial-frequency joint modality contrastive learning scheme is designed to directly decouple the modality-specific information from MRI data. To decompose tumor-specific representations and extract discriminative holistic features, we propose an affinity-guided dense tumor-region knowledge distillation mechanism in the TD-Stage through aligning the features of a disentangled binary teacher network with a holistic student network. By explicitly discovering relations among modalities and tumor regions, our model can learn sufficient information for segmentation even if some modalities are missing. Extensive experiments on the public BraTS-2018 database demonstrate the superiority of our framework over state-of-the-art methods in missing modalities situations. Codes are available at https://github.com/CityU-AIM-Group/D2Net.

Source free domain adaptation for medical image segmentation with fourier style mining.

Unsupervised domain adaptation (UDA) aims to exploit the knowledge learned from a labeled source dataset to solve similar tasks in a new unlabeled target domain. Existing UDA techniques typically assume that samples from source and target domains are freely accessible during the training. However, it may be impractical to access source images due to privacy concerns, especially in medical imaging scenarios with the patient information. To tackle this issue, we devise a novel source free domain adaptation framework with fourier style mining, where only a well-trained source segmentation model is available for the adaptation to the target domain. Our framework is composed of two stages: a generation stage and an adaptation stage. In the generation stage, we design a Fourier Style Mining (FSM) generator to inverse source-like images through statistic information of the pretrained source model and mutual Fourier Transform. These generated source-like images can provide source data distribution and benefit the domain alignment. In the adaptation stage, we design a Contrastive Domain Distillation (CDD) module to achieve feature-level adaptation, including a domain distillation loss to transfer relation knowledge and a domain contrastive loss to narrow down the domain gap by a self-supervised paradigm. Besides, a Compact-Aware Domain Consistency (CADC) module is proposed to enhance consistency learning by filtering out noisy pseudo labels with shape compactness metric, thus achieving output-level adaptation. Extensive experiments on cross-device and cross-centre datasets are conducted for polyp and prostate segmentation, and our method delivers impressive performance compared with state-of-the-art domain adaptation methods. The source code is available at https://github.com/CityU-AIM-Group/SFDA-FSM.

A Source-Free Domain Adaptive Polyp Detection Framework With Style Diversification Flow.

The automatic detection of polyps across colonoscopy and Wireless Capsule Endoscopy (WCE) datasets is crucial for early diagnosis and curation of colorectal cancer. Existing deep learning approaches either require mass training data collected from multiple sites or use unsupervised domain adaptation (UDA) technique with labeled source data. However, these methods are not applicable when the data is not accessible due to privacy concerns or data storage limitations. Aiming to achieve source-free domain adaptive polyp detection, we propose a consistency based model that utilizes Source Model as Proxy Teacher (SMPT) with only a transferable pretrained model and unlabeled target data. SMPT first transfers the stored domain-invariant knowledge in the pretrained source model to the target model via Source Knowledge Distillation (SKD), then uses Proxy Teacher Rectification (PTR) to rectify the source model with temporal ensemble of the target model. Moreover, to alleviate the biased knowledge caused by domain gaps, we propose Uncertainty-Guided Online Bootstrapping (UGOB) to adaptively assign weights for each target image regarding their uncertainty. In addition, we design Source Style Diversification Flow (SSDF) that gradually generates diverse style images and relaxes style-sensitive channels based on source and target information to enhance the robustness of the model towards style variation. The capacities of SMPT and SSDF are further boosted with iterative optimization, constructing a stronger framework SMPT++ for cross-domain polyp detection. Extensive experiments are conducted on five distinct polyp datasets under two types of cross-domain settings. Our proposed method shows the state-of-the-art performance and even outperforms previous UDA approaches that require the source data by a large margin. The source code is available at github.com/CityU-AIM-Group/SFPolypDA.

Non-equivalent images and pixels: Confidence-aware resampling with meta-learning mixup for polyp segmentation.

Automatic segmentation of polyp regions in endoscope images is essential for the early diagnosis and surgical planning of colorectal cancer. Recently, deep learning-based approaches have achieved remarkable progress for polyp segmentation, but they are at the expense of laborious large-scale pixel-wise annotations. In addition, these models treat samples equally, which may cause unstable training due to polyp variability. To address these issues, we propose a novel Meta-Learning Mixup (MLMix) data augmentation method and a Confidence-Aware Resampling (CAR) strategy for polyp segmentation. MLMix adaptively learns the interpolation policy for mixup data in a data-driven way, thereby transferring the original soft mixup label to a reliable hard label and enriching the limited training dataset. Considering the difficulty of polyp image variability in segmentation, the CAR strategy is proposed to progressively select relatively confident images and pixels to facilitate the representation ability of model and ensure the stability of the training procedure. Moreover, the CAR strategy leverages class distribution prior knowledge and assigns different penalty coefficients for polyp and normal classes to rebalance the selected data distribution. The effectiveness of the proposed MLMix data augmentation method and CAR strategy is demonstrated through comprehensive experiments, and our proposed model achieves state-of-the-art performance with 87.450% dice on the EndoScene test set and 86.453% dice on the wireless capsule endoscopy (WCE) polyp dataset.

Mutual-Prototype Adaptation for Cross-Domain Polyp Segmentation.

Accurate segmentation of the polyps from colonoscopy images provides useful information for the diagnosis and treatment of colorectal cancer. Despite deep learning methods advance automatic polyp segmentation, their performance often degrades when applied to new data acquired from different scanners or sequences (target domain). As manual annotation is tedious and labor-intensive for new target domain, leveraging knowledge learned from the labeled source domain to promote the performance in the unlabeled target domain is highly demanded. In this work, we propose a mutual-prototype adaptation network to eliminate domain shifts in multi-centers and multi-devices colonoscopy images. We first devise a mutual-prototype alignment (MPA) module with the prototype relation function to refine features through self-domain and cross-domain information in a coarse-to-fine process. Then two auxiliary modules: progressive self-training (PST) and disentangled reconstruction (DR) are proposed to improve the segmentation performance. The PST module selects reliable pseudo labels through a novel uncertainty guided self-training loss to obtain accurate prototypes in the target domain. The DR module reconstructs original images jointly utilizing prediction results and private prototypes to maintain semantic consistency and provide complement supervision information. We extensively evaluate the proposed model in polyp segmentation performance on three conventional colonoscopy datasets: CVC-DB, Kvasir-SEG, and ETIS-Larib. The comprehensive experimental results demonstrate that the proposed model outperforms state-of-the-art methods.

Consolidated domain adaptive detection and localization framework for cross-device colonoscopic images.

Automatic polyp detection has been proven to be crucial in improving the diagnosis accuracy and reducing colorectal cancer mortality during the precancerous stage. However, the performance of deep neural networks may degrade severely when being deployed to polyp data in a distinct domain. This domain distinction can be caused by different scanners, hospitals, or imaging protocols. In this paper, we propose a consolidated domain adaptive detection and localization framework to bridge the domain gap between different colonosopic datasets effectively, consisting of two parts: the pixel-level adaptation and the hierarchical feature-level adaptation. For the pixel-level adaptation part, we propose a Gaussian Fourier Domain Adaptation (GFDA) method to sample the matched source and target image pairs from Gaussian distributions then unify their styles via the low-level spectrum replacement, which can reduce the domain discrepancy of the cross-device polyp datasets in appearance level without distorting their contents. The hierarchical feature-level adaptation part comprising a Hierarchical Attentive Adaptation (HAA) module to minimize the domain discrepancy in high-level semantics and an Iconic Concentrative Adaptation (ICA) module to perform reliable instance alignment. These two modules are regularized by a Generalized Consistency Regularizer (GCR) for maintaining the consistency of their domain predictions. We further extend our framework to the polyp localization task and present a Centre Besiegement (CB) loss for better location optimization. Experimental results show that our framework outperforms other domain adaptation detectors by a large margin in the detection task meanwhile achieves the state-of-the-art recall rate of 87.5% in the localization task. The source code is available at https://github.com/CityU-AIM-Group/ConsolidatedPolypDA.

Learn to Threshold: ThresholdNet With Confidence-Guided Manifold Mixup for Polyp Segmentation.

The automatic segmentation of polyp in endoscopy images is crucial for early diagnosis and cure of colorectal cancer. Existing deep learning-based methods for polyp segmentation, however, are inadequate due to the limited annotated dataset and the class imbalance problems. Moreover, these methods obtained the final polyp segmentation results by simply thresholding the likelihood maps at an eclectic and equivalent value (often set to 0.5). In this paper, we propose a novel ThresholdNet with a confidence-guided manifold mixup (CGMMix) data augmentation method, mainly for addressing the aforementioned issues in polyp segmentation. The CGMMix conducts manifold mixup at the image and feature levels, and adaptively lures the decision boundary away from the under-represented polyp class with the confidence guidance to alleviate the limited training dataset and the class imbalance problems. Two consistency regularizations, mixup feature map consistency (MFMC) loss and mixup confidence map consistency (MCMC) loss, are devised to exploit the consistent constraints in the training of the augmented mixup data. We then propose a two-branch approach, termed ThresholdNet, to collaborate the segmentation and threshold learning in an alternative training strategy. The threshold map supervision generator (TMSG) is embedded to provide supervision for the threshold map, thereby inducing better optimization of the threshold branch. As a consequence, ThresholdNet is able to calibrate the segmentation result with the learned threshold map. We illustrate the effectiveness of the proposed method on two polyp segmentation datasets, and our methods achieved the state-of-the-art result with 87.307% and 87.879% dice score on the EndoScene dataset and the WCE polyp dataset. The source code is available at https://github.com/Guo-Xiaoqing/ThresholdNet.

Semi-supervised WCE image classification with adaptive aggregated attention.

Accurate abnormality classification in Wireless Capsule Endoscopy (WCE) images is crucial for early gastrointestinal (GI) tract cancer diagnosis and treatment, while it remains challenging due to the limited annotated dataset, the huge intra-class variances and the high degree of inter-class similarities. To tackle these dilemmas, we propose a novel semi-supervised learning method with Adaptive Aggregated Attention (AAA) module for automatic WCE image classification. Firstly, a novel deformation field based image preprocessing strategy is proposed to remove the black background and circular boundaries in WCE images. Then we propose a synergic network to learn discriminative image features, consisting of two branches: an abnormal regions estimator (the first branch) and an abnormal information distiller (the second branch). The first branch utilizes the proposed AAA module to capture global dependencies and incorporate context information to highlight the most meaningful regions, while the second branch mainly focuses on these calculated attention regions for accurate and robust abnormality classification. Finally, these two branches are jointly optimized by minimizing the proposed discriminative angular (DA) loss and Jensen-Shannon divergence (JS) loss with labeled data as well as unlabeled data. Comprehensive experiments have been conducted on the public CAD-CAP WCE dataset. The proposed method achieves 93.17% overall accuracy in a fourfold cross-validation, verifying its effectiveness for WCE image classification. The source code is available at https://github.com/Guo-Xiaoqing/SSL_WCE.

Deep learning for identification of critical regions associated with toxicities after liver stereotactic body radiation therapy.

PURPOSE: Radiation therapy (RT) is prescribed for curative and palliative treatment for around 50% of patients with solid tumors. Radiation-induced toxicities of healthy organs accompany many RTs and represent one of the main limiting factors during dose delivery. The existing RT planning solutions generally discard spatial dose distribution information and lose the ability to recognize radiosensitive regions of healthy organs potentially linked to toxicity manifestation. This study proposes a universal deep learning-based algorithm for recognitions of consistent dose patterns and generation of toxicity risk maps for the abdominal area. METHODS: We investigated whether convolutional neural networks (CNNs) can automatically associate abdominal computed tomography (CT) images and RT dose plans with post-RT toxicities without being provided segmentation of abdominal organs. The CNNs were also applied to study RT plans, where doses at specific anatomical regions were reduced/increased, with the aim to pinpoint critical regions sparing of which significantly reduces toxicity risks. The obtained risk maps were computed for individual anatomical regions inside the liver and statistically compared to the existing clinical studies. RESULTS: A database of 122 liver stereotactic body RT (SBRT) executed at Stanford Hospital from July 2004 and November 2015 was assembled. All patients treated for primary liver cancer, mainly hepatocellular carcinoma and cholangiocarcinoma, with complete follow-ups were extracted from the database. The SBRT treatment doses ranged from 26 to 50 Gy delivered in 1-5 fractions for primary liver cancer. The patients were followed up for 1-68 months depending on the survival time. The CNNs were trained to recognize acute and late grade 3+ biliary stricture/obstruction, hepatic failure or decompensation, hepatobiliary infection, liver function test (LFT) elevation or/and portal vein thrombosis, named for convenience hepatobiliary (HB) toxicities. The toxicity prediction accuracy was of 0.73 measured in terms of the area under the receiving operator characteristic curve. Significantly higher risk scores (P < 0.05) of HB toxicity manifestation were associated with irradiation for the hepatobiliary tract in comparison to the risk scores for liver segments I-VIII and portal vein. This observation is in strong agreement with anatomical and clinical expectations. CONCLUSION: In this work, we proposed and validated a universal deep learning-based solution for the identification of radiosensitive anatomical regions. Without any prior anatomical knowledge, CNNs automatically recognized the importance of hepatobiliary tract sparing during liver SBRT.

Multi-subtype classification model for non-small cell lung cancer based on radiomics: SLS model.

PURPOSE: Histological subtypes of non-small cell lung cancer (NSCLC) are crucial for systematic treatment decisions. However, the current studies which used noninvasive radiomic methods to classify NSCLC histology subtypes mainly focused on two main subtypes: squamous cell carcinoma (SCC) and adenocarcinoma (ADC), while multi-subtype classifications that included the other two subtypes of NSCLC: large cell carcinoma (LCC) and not otherwise specified (NOS), were very few in the previous studies. The aim of this work was to establish a multi-subtype classification model for the four main subtypes of NSCLC and improve the classification performance and generalization ability compared with previous studies. METHODS: In this work, we extracted 1029 features from regions of interest in computed tomography (CT) images of 349 patients from two different datasets using radiomic methods. Based on "three-in-one" concept, we proposed a model called SLS wrapping three algorithms, synthetic minority oversampling technique, ℓ2,1-norm minimization, and support vector machines, into one hybrid technique to classify the four main subtypes of NSCLC: SCC, ADC, LCC, and NOS, which could cover the whole range of NSCLC. RESULTS: We analyzed the 247 features obtained by dimension reduction, and found that the extracted features from three methods: first order statistics, gray level co-occurrence matrix, and gray level size zone matrix, were more conducive to the classification of NSCLC subtypes. The proposed SLS model achieved an average classification accuracy of 0.89 on the training set (95% confidence interval [CI]: 0.846 to 0.912) and a classification accuracy of 0.86 on the test set (95% CI: 0.779 to 0.941). CONCLUSIONS: The experiment results showed that the subtypes of NSCLC could be well classified by radiomic method. Our SLS model can accurately classify and diagnose the four subtypes of NSCLC based on CT images, and thus it has the potential to be used in the clinical practice to provide valuable information for lung cancer treatment and further promote the personalized medicine.

Neural Networks for Deep Radiotherapy Dose Analysis and Prediction of Liver SBRT Outcomes.

Stereotactic body radiation therapy (SBRT) is a relatively novel treatment modality, with little post-treatment prognostic information reported. This study proposes a novel neural network based paradigm for accurate prediction of liver SBRT outcomes. We assembled a database of patients treated with liver SBRT at our institution. Together with a three-dimensional (3-D) dose delivery plans for each SBRT treatment, other variables such as patients' demographics, quantified abdominal anatomy, history of liver comorbidities, other liver-directed therapies, and liver function tests were collected. We developed a multi-path neural network with the convolutional path for 3-D dose plan analysis and fully connected path for other variables analysis, where the network was trained to predict post-SBRT survival and local cancer progression. To enhance the network robustness, it was initially pre-trained on a large database of computed tomography images. Following n-fold cross-validation, the network automatically identified patients that are likely to have longer survival or late cancer recurrence, i.e., patients with the positive predicted outcome (PPO) of SBRT, and vice versa, i.e., negative predicted outcome (NPO). The predicted results agreed with actual SBRT outcomes with 56% of PPO patients and 0% NPO patients with primary liver cancer survived more than two years after SBRT. Similarly, 82% of PPO patients and 0% of NPO patients with metastatic liver cancer survived two-year threshold. The obtained results were superior to the performance of support vector machine and random forest classifiers. Furthermore, the network was able to identify the critical-to-spare liver regions, and the critical clinical features associated with the highest risks of negative SBRT outcomes.

Prostate cancer classification with multiparametric MRI transfer learning model.

PURPOSE: Prostate cancer classification has a significant impact on the prognosis and treatment planning of patients. Currently, this classification is based on the Gleason score analysis of biopsied tissues, which is neither accurate nor risk free. This study aims to learn discriminative features in prostate images and assist physicians in classifying prostate cancer automatically. METHODS: We develop a novel multiparametric magnetic resonance transfer learning (MPTL) method to automatically stage prostate cancer. We first establish a deep convolutional neural network with three branch architectures, which transfer pretrained model to compute features from multiparametric MRI images (mp-MRI): T2w transaxial, T2w sagittal, and apparent diffusion coefficient (ADC). The learned features are concatenated to represent information of mp-MRI sequences. A new image similarity constraint is then proposed to enable the distribution of the features within the same category in a narrow angle region. With the joint constraints of softmax loss and image similarity loss in the fine-tuning process, the MPTL can provide descriptive features with intraclass compactness and interclass separability. RESULTS: Two cohorts: 132 cases from our institutional review board-approved patient database and 112 cases from the PROSTATEx-2 Challenge are utilized to evaluate the robustness and effectiveness of the proposed MPTL model. Our model achieved high accuracy of prostate cancer classification (accuracy of 86.92%). Moreover, the comparison results demonstrate that our method outperforms both hand-crafted feature-based methods and existing deep learning models in prostate cancer classification with higher accuracy. CONCLUSION: The experiment results showed that the proposed method can learn discriminative features in prostate images and classify the cancer accurately. Our MPTL model could be further applied in the clinical practice to provide valuable information for cancer treatment and precision medicine.

Development of deep neural network for individualized hepatobiliary toxicity prediction after liver SBRT.

BACKGROUND: Accurate prediction of radiation toxicity of healthy organs-at-risks (OARs) critically determines the radiation therapy (RT) success. The existing dose-volume histogram-based metric may grossly under/overestimate the therapeutic toxicity after 27% in liver RT and 50% in head-and-neck RT. We propose the novel paradigm for toxicity prediction by leveraging the enormous potential of deep learning and go beyond the existing dose/volume histograms. EXPERIMENTAL DESIGN: We employed a database of 125 liver stereotactic body RT (SBRT) cases with follow-up data to train deep learning-based toxicity predictor. Convolutional neural networks (CNNs) were applied to discover the consistent patterns in 3D dose plans associated with toxicities. To enhance the predicting power, we first pretrain the CNNs with transfer learning from 3D CT images of 2644 human organs. CNNs were then trained on liver SBRT cases. Furthermore, nondosimetric pretreatment features, such as patients' demographics, underlying liver diseases, liver-directed therapies, were inputted into the fully connected neural network for more comprehensive prediction. The saliency maps of CNNs were used to estimate the toxicity risks associated with irradiation of anatomical regions of specific OARs. In addition, we applied machine learning solutions to map numerical pretreatment features with hepatobiliary toxicity manifestation. RESULTS: Among 125 liver SBRT patients, 58 were treated for liver metastases, 36 for hepatocellular carcinoma, 27 for cholangiocarcinoma, and 4 for other histologies. We observed that CNN we able to achieve accurate hepatobiliary toxicity prediction with the AUC of 0.79, whereas combining CNN for 3D dose plan analysis and fully connected neural networks for numerical feature analysis resulted in AUC of 0.85. Deep learning produces almost two times fewer false-positive toxicity predictions in comparison to DVH-based predictions, when the number of false negatives, i.e., missed toxicities, was minimized. The CNN saliency maps automatically estimated the toxicity risks for portal vein (PV) regions. We discovered that irradiation of the proximal portal vein is associated with two times higher toxicity risks (risk score: 0.66) that irradiation of the left portal vein (risk score: 0.31). CONCLUSIONS: The framework offers clinically accurate tools for hepatobiliary toxicity prediction and automatic identification of anatomical regions that are critical to spare during SBRT.